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Abstract 

Engineering and technology education (ETE) are receiving increased 
attention as components of STEM education. Curriculum development should 
be informed by perceptions of academic engineering educators (AEEs) and 
classroom technology teachers (CTTs) as both groups educate students to 
succeed in the technological world. The purpose of this study was to identify 
ETE concepts and skills needed by all high school students in the United States 
and to compare perceptions of AEEs and CTTs relative to their importance. This 
research was carried out using a modified Delphi research methodology 
involving three survey rounds interspersed with controlled opinion feedback. 

Consensus was found on 14 of 38 survey items within five ETE domains 
(design, modeling, systems, resources, and human values) that are repeatedly 
referenced in the literature. The most important competencies for high school 
students to learn were to: (1) identify and discuss environmental, health, and safety 
issues; (2) use representational modeling to convey the essence of a design; (3) use 
verbal or visual means to explain why an engineering design decision was made; and 
(4) show evidence of considering human factors when proposing design solutions. 
The study established a consensus between AEEs and CTTs that contributes to 
the body of knowledge about what high school students should learn in ETE. 
Study results can inform curriculum development and revision of the Standards 
for Technological Literacy. 

Keywords: Delphi; engineering and technology education; high school; STEM; 
survey 

Because of the essential roles that engineering and technology play in 
addressing societal and environmental challenges, support for PreK-12 
engineering and technology education (ETE) programs in the United States has 
been rapidly growing (Katehi, Pearson, & Feder, 2009). There is growing 
recognition that school-based ETE experiences can be pedagogically valuable for 
all students—not only in providing an effective way to contextualize and reinforce 
STEM skills but also in mobilizing engineering thinking as a way for young people 
to approach problems of all kinds (Brophy & Evangelou, 2007; Forlenza, 2010). 
The purpose of this study was to compare the perceptions of two constituencies 
whose missions focus on preparing students to succeed in our technological world 
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through engineering and technology education: academic engineering educators 
(AEEs) who prepare future engineers at the university level and high school 
classroom technology teachers (CTTs) who teach engineering and technology 
courses at the secondary school level. The study established a consensus among the 
groups about the most important ETE concepts and skills that all students in the 
United States should learn by the time they graduate from high school. 

Literature Review 

A literature review established a basis for identifying competencies for the 
initial item set in the study’s survey instrument. The review also determined how to 
optimally use Delphi research methodology to converge expert opinion to arrive at 
consensus (RAND Corporation, 2017) and examined differences between 
engineering and technology and the preparation of professionals in those fields. 

Differences Between Engineering and Technology Engineering. 

According to the Engineers’ Council for Professional Development (ECPD), the 
predecessor of the Accreditation Board for Engineering and Technology 
(ABET): 

Engineering is the profession in which a knowledge of the mathematical 
and natural sciences gained by study, experience and practice is applied 
with judgment to develop ways to utilize, economically, the materials and 
forces of nature for the benefit of mankind. (ECPD, 1979; as cited in 
National Research Council [NRC], 1986, p. 74) 


An earlier (1941) definition advanced by ECPD the is that “engineering is the 
creative application of scientific principles to design or develop structures, 
machines, apparatus, or manufacturing processes, or works utilizing them singly 
or in combination” (Greefhorst & Proper, 2011, p. 9). Bloch (1986) wrote that 
“engineering is the process of investigating how to solve problems. This process 
leads to a body of engineering knowledge consisting of concepts, methods, data 
bases, and, frequently, physical expressions of results” (p. 28). Wulf and Fisher 
(2002) describe “what engineers do . . . [as] ‘design under constraint’” (p. 36). 

Technology. The National Assessment Governing Board (2013) defines 
technology as “any modification of the natural world done to fulfill human 
needs or desires” (p. xi). According to de Vries (2005), technology is “the 
human activity that transforms the natural environment to make it fit better with 
human needs, thereby using various kinds of information and knowledge, 
various kinds of natural (material, energy) and cultural resources (money, social 
relationships, etc.)” (p. 11). Kline (1985) suggests that technology is viewed in 
four ways: as an artifact, as a methodology or technique, as a system of 
production, and as a sociotechnical system. Swyt (1989), at the National 
Institute of Science and Technology, differentiates between engineering and 
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technology by explaining that engineering is oriented toward the solution of 
specific problems, whereas technology is oriented toward development of new 
capability. 

Preparation of Academic Engineering Educators and Classroom Technology 
Teachers 

Academic engineering educators and classroom technology teachers in the 
United States come from different educational traditions, although both groups 
advocate the importance of technological literacy for the general population. 
Engineering emerged as a separate subject with the founding of the first schools 
of engineering and professional societies in the 18th century. AEEs typically 
have postgraduate degrees in engineering. In the United States, technology 
education emerged from industrial arts, and worldwide, technology education had 
its roots in crafts teaching. State-certified CTTs typically have master’s degrees 
in technology education. 

ABET Program Standards for Engineering Programs. The Criteria for 
Accrediting Engineering Programs by ABET’s Engineering Accreditation 
Commission (2012) state that engineering graduates must have the ability to 
“apply knowledge of mathematics, science, and engineering”; “design and 
conduct experiments, as well as analyze and interpret data”; “design a system, 
component, or process to meet desired needs”; “function on multidisciplinary 
teams”; “identify, formulate, and solve engineering problems”; “communicate 
effectively”; and “use the techniques, skills, and tools necessary for engineering 
practice” (p. 3). Graduates must also “understand the impact of engineering 
solutions in a global, economic, environmental, and societal context”; recognize 
“the need for, and an ability to engage in life-long learning”; and understand 
“contemporary issues” (p. 3). ABET requires educational programs to include a 
major engineering design experience that builds upon the fundamental concepts 
of mathematics, basic sciences, the humanities and social sciences, engineering 
topics, and communication skills. Engineering topics must include subjects in 
the engineering sciences and engineering design, which “have their roots in 
mathematics and basic sciences but carry knowledge further toward creative 
application” (p. 4). 

NCATE Program Standards for Technology Education Programs. The 

National Council for Accreditation of Teacher Education (NCATE) is the 
education profession’s mechanism to help establish high-quality teacher 
preparation programs (National Council for Accreditation of Teacher Education 
[NCATE], 2008). NCATE has developed program standards (e.g., International 
Technology Education Association [ITEA] Council on Technology Teacher 
Education [CTTE], 2003; NCATE, 2008) that define the criteria for accrediting 
technology education programs in much the same manner as ABET has defined 
criteria for accrediting engineering programs. NCATE standards (ITEA, CTTE, 
2003) state that “within the contexts of the Designed World ,” “technology 
teacher education program candidates [must] develop an understanding of the 
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nature of technology” (p. 22), “of technology and society” (p. 24), “of design” 

(p. 26), and “of the designed world” (p. 30) as well as “develop abilities for a 
technological world” (p. 28). Candidates must also “design, implement, and 
evaluate curricula based upon Standards for Technological Literacy” (p. 32), 
“use a variety of effective teaching practices that enhance learning of 
technology” (p. 34), “design, create, and manage learning environments that 
promote technological literacy” (p. 36), “understand students as learners, and 
how commonality and diversity affect learning” (p. 38), and engage “in 
comprehensive and sustained professional growth” (p. 40). 

In a comparison of professional competencies required by ABET for engineers 
and NCATE for technology teachers, Hacker (2005) found that ABET focused on 
technical content preparation for engineers, whereas NCATE focused on pedagogy 
for teachers; however, a high degree of alignment was evident with respect to other 
competencies. He also found that both professional groups were well prepared in 
areas of professional practice, design and problem solving, team functioning, 
ethical and professional responsibility, communication skills, social and cultural 
impacts, and professional growth. One clear difference between the groups was that 
engineers were much more rigorously prepared in mathematics and science than 
technology teachers. 

Projects Oriented Toward Formulating an ETE Knowledge and Skill Base 

Major projects that have identified student learning outcomes in ETE 
include the Standards for Technological Literacy (STL; International 
Technology Education Association [1TEA], 2007); the National Academy of 
Engineering (NAE) reports (Katehi, Pearson, & Feder, 2009; National Academy 
of Engineering [NAE], 2010); the National Research Council’s (2012) 
Framework for K—l 2 Science Education and the Next Generation Science 
Standards (NGSS Lead States, 2013), which built upon it; the Technology’ and 
Engineering Literacy Framework for the National Assessment of Educational 
Progress (NAEP; National Assessment Governing Board [NAGB], 2013); and 
studies conducted by Custer, Daugherty, and Meyer (2010), Childress and 
Rhodes (2008), and Rossouw, Hacker, and de Vries (2010). 

The International Technology Education Association (ITEA), now the 
International Technology and Engineering Educators Association (ITEEA), 
developed the STL to identify “what students should know and be able to do in 
order to be technologically literate” (ITEA, 2007, p. vii). The standards are divided 
into five knowledge categories (comprising 20 content standards and 98 
benchmarks at the Grades 9-12 level): the nature of technology’, technology’ and 
society, design, abilities for a technological world, and the designed world. 

NAE’s Committee on Standards for K-12 Engineering Education (2010) 
reviewed eight prior studies “that attempted] to identify of core concepts, skills, and 
dispositions appropriate to K-12 engineering education” (p. 24). The 16 categories 
that they found included: design, STEM connections, engineering and society, 
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constraints, communication, systems, systems thinking, modeling, optimization, 
analysis, collaboration and teamwork, creativity, knowledge of specific 
technologies, nature of engineering, prototyping, and experimentation (p. 35). 

The Next Generation Science Standards (NGSS) (NGSS Lead States, 2013) 
grew from the National Research Council’s (2012) Framework for K-12 Science 
Education. The NGSS integrated disciplinary’ core ideas, science and 
engineering practices, and crosscutting concepts related to technology and 
engineering (including design, modeling, and systems) into student performance 
expectations (NGSS Lead States, 2013). 

The National Assessment of Educational Progress (NAEP) is an assessment 
of “what U.S. students know and are able to do in a range of subject areas” 
(NAGB, 2013, p. ix). In 2014, the NAEP Technology and Engineering Literacy 
Assessment was administered to 21,500 students in Grade 8 (The Nation’s 
Report Card, 2016). “The assessment. . . consists] of technological content 
areas . . . and technological practices that characterize the field” among which 
are design and systems, information and communication technology, and 
technology and society (NAGB, 2013, p. A-9). 

In a study titled “Formulating a Concept Base for Secondary Level 
Engineering: A Review and Synthesis,” Custer, Daugherty, and Meyer (2010) 
identified 13 major engineering concepts (among them design, systems, and 
modeling) that were drawn from a variety of sources in the literature and from 
three focus groups of engineering experts. 

In another study, Childress and Rhodes (2008) examined what high school 
students “should know and be able to do prior to entry into a postsecondary 
engineering program” (p. 5). Categories identified included engineering design, 
applications of engineering design, engineering analysis, engineering and human 
values, engineering communication, engineering science, and emerging fields of 
engineering. 

As a part of the Concepts and Contexts in Engineering and Technology 
Education (CCETE) Project, a collaboration between Delft University of 
Technology in the Netherlands and Flofstra University in New York State, Rossouw 
et al. (2010) conducted a Delphi study with 32 international experts from nine 
countries to identify overarching themes and contexts that could be used to develop 
curricula for education about engineering and technology was developed. Table 1 
lists the five main themes and associated subconcepts identified in that study. 
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Table 1 

CCETE Project Overarching Themes and Subconcepts 

Themes_Subconcepts 

Design Optimization and trade-offs; criteria and constraints; iteration 

Modeling Representational, explanatory, predictive 

Systems Systems/subsystems; input-process-output; feedback and control 

Resources Materials, energy, information, time, tools, humans, capital 
Human values Sustainability; technological assessment; creativity/innovation; 
ethical decisions 


In this comparison of perceptions study, we used the five themes that emerged 
from the CCETE Project study as organizing categories because they aligned so well 
with those identified by other major projects. Further details about important ETE 
concepts and skills within these categories were added. 

Summary of the Literature Review 

Through the literature review, we identified ETE knowledge and skill sets that 
scholars believe to be important for all high school students to learn within their 
fundamental education. These concepts and skills informed the set of items that 
comprised this study’s Round 1 survey instrument. We established the basis 
upon which expert panelists suggested additions, changes, or deletions to survey 
items in subsequent Delphi rounds. 

Research Questions 

The research questions for this comparison of perceptions study were: 

1. RQ1: Where does the strongest consensus exist among the expert 
panelists relative to the importance of specific ETE concepts and skills 
that all high school students in the United States should attain as part of 
their fundamental education? 

2. RQ2: Which ETE concepts and skills does the expert panel perceive to 
be most important for high school students to attain as part of their 
fundamental education? 

3. RQ3: Where are there significant differences between academic 
engineering educators’ and classroom technology teachers’ perceptions 
of the importance of ETE concepts and skills? 

4. RQ4: Which concepts and skills that academic engineering educators 
and classroom technology teachers agree are highly important are not 
presently addressed by the STL? 
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Methodology 

In this study, we employed Delphi survey research methodology because it is 
effective in soliciting and converging experts’ opinions to obtain consensus 
(Salancik, Wenger, & Heifer, 1971). Delphi methodology assures anonymity, 
provides ongoing feedback to participants, and reduces the effects of bias due to 
group interaction (Dalkey, 1972). 

The purpose of a Delphi study is “to obtain the most reliable opinion 
consensus of a group of experts by subjecting them to a series of intensive 
questionnaires in depth interspersed with controlled opinion feedback” (Dalkey & 
Helmer, 1963, p. v). Studies comparing Delphi with other methods (Ulschak, 

1983) confirmed its effectiveness in generating ideas and its efficient use 
participants’ time. 

Typically, a Delphi study starts by asking participants to respond to a 
specific question or issue. In subsequent rounds, participants are asked to 
consider feedback from the previous round, and the instrument is modified to 
reflect experts’ opinions. “The essential feature is the use of quantitative 
feedback given to each participant” (Uebersax, 2000, 4.1 The Delphi Method, 
para. 1). When respondents’ estimates for an item do not fall within the range of 
group responses, they are asked to reconsider their position and, when justified, 
change their response. Thus, an attempt is made to achieve consensus (Wicklein 
&Rojewski, 1999). 

As is often done in Delphi studies (Chalmers, 2014; Greer, 2008; Iqbal & 
Pipon-Young, 2009; Scott, Washer, & Wright, 2006), we used open-ended text 
boxes to invite panel members to provide feedback during survey rounds and at 
the conclusion of the survey. 

Modified Delphi Methodology 

In this study, we used a modified Delphi research methodology, which “is 
similar to the full Delphi in terms of procedure (i.e., a series of rounds with selected 
experts) and intent (i.e., to predict future events and arrive at consensus)” (Custer, 
Scarcella, & Stewart, 1999, p. 51). Modifications included: (a) “beginning the 
process with a set of pre-selected items” (p. 51) that were drawn from the literature 
review and validated by experts and (b) adding validation panel meetings. Starting 
with a set of preselected items “(a) typically improves the initial round response 
rate, and (b) provides a solid grounding in previously developed work” (p. 51). 
Meetings of a validation panel verified the importance and level of abstraction of 
initial items, vetted prospective panelists to confirm their expertise, and added 
structure to the survey (Rossouw, Hacker, & de Vries, 2010). 

In accordance with the method suggested by Fowles (1978), seven stages 
characterized this study’s Delphi procedure. 

• Stage 1: Define the research questions. 

• Stage 2: Assemble the panel of experts (with help from the validation 
panel). 
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• Stage 3: Design and validate the initial set of survey items (with 
validation panel help). 

• Stage 4: Conduct the three-round Delphi survey. 

• Round 1 included a beginning set of concepts drawn from the 
literature review. 

• Round 2 reflected changes based on panel input and solicited 
additional suggestions. 

• Round 3 included further changes based on final panel review. 

• Stage 5: Analyze survey results. 

• Stage 6: Summarize Conclusions. 

• Stage 7: Convene validation panel to review researchers’ conclusions 
and reach consensus. 

In the literature, three Delphi rounds have been found sufficient to arrive at 
consensus (Brooks, 1979) because after three iterations, not enough new 
information is gained to warrant the cost of more administrations (Altschuld, 
1993). Panelists were asked to rate each concept on a 7-point Likert scale using 
these descriptors: strongly agree (7), 6, moderately agree (6), agree (5), 
indifferent (4), moderately disagree (3), disagree (2), or strongly disagree (1). 
Panelists were invited to suggest and justify items that should be added or 
deleted. Panelists were informed that items would be modified based on their 
suggestions, and they were invited to reconsider item ratings if theirs were at 
variance with whole-group median ratings. 

Participant Selection and Panel Size 

Because the success of the Delphi technique relies upon experts’ judgment, 
selection of panelists was critical, and random selection was not considered. “Large 
numbers of respondents generate many items and ideas making the summarizing 
process difficult” (Ludwig, 1997, Participation Selection, para. 1). Delbecq, Van 
deVen, and Gustafson (1975) suggest that 10 to 15 panelists are sufficient. Dalkey 
(1972) reported that reliability, with a correlation coefficient approaching 0.9, was 
found with a panel size of 13. J. G. Wells (personal communication, March 9, 

2013) suggested that in research concerned with intragroup and intergroup 
judgments, subgroups of 16 panelists should be recruited. To allow for attrition, we 
recruited 18 AEEs and 17 CTTs (35 panel members in total) for this study. 

Selection Criteria 

Participants were selected because they were leading authorities in their 
fields with (a) documented participation in initiatives linking engineering and 
K-12 education, (b) a minimum of 5 years of experience teaching engineering 
or technology education, and (c) proven ability to formulate their thinking 
through research or active involvement in major funded projects. Participants 
were identified through recommendations from professional organizations and 
agencies (the American Society of Engineering Education, 1TEEA, NAE, the 
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National Science Foundation, and the New York State Technology and 
Engineering Educators Association) and recommendations from validation panel 
members. 

Validation Panel 

The validation panel was composed of the researchers, two AEEs with over 
10 years of K-12 ETE experience, and two CTTs who are professional leaders 
with over 10 years of K-12 ETE experience. Validation panel meetings were 3 
hours in duration. A meeting was held at the onset of the study to assist us in 
selecting panelists and validating survey items. The second meeting was held after 
the study concluded to discuss results, frame conclusions, and establish a cutoff 
point for items to be deemed as highly important for all high school students to 
learn. 

Instrumentation and Data Analysis Methodology 

The survey was tested and conducted online using Qualtrics (2014) survey 
software. Data was exported to SPSS (Version 22.0) for analysis. With Likert scale 
data, the use of median scores is strongly favored (Hill & Fowles, 1975; Eckrnan, 
1983; Jacobs, 1996). Data were treated as ordinal data (Comrey, 1973) and were 
reported using descriptive statistics: medians, frequencies, percentiles, and 
interquartile range (IQR) statistics. A nonparametric test (the Mann-Whitney U) 
was used to determine statistically significant differences between the two study 
groups, and /^-values were reported at the a = 0.05 level. Data provided insight 
into the study’s research questions as follows: 

Determining consensus (Research Question 1). Data analysis determined the 
strength of consensus on each item by subgroup and whole group. According to 
Rojewski and Meers (1991), “Consensus ... [is] determined using the interquartile 
range of each research priority [or concept] statement. Interquartile Range is a 
descriptive statistic defined as the distance between the first and third quartiles (i.e., 
the middle 50% of scores)” (p. 36). Low IQRs are one measure of strong consensus 
on an item. 

In this study, we used a 7-point scale, and whole-group IQRs ranged from 
0.79 to 1.98. After an analysis of scores within each quartile for each item, the 
researchers and the validation panel established that an IQR of < 1.61 should be 
considered an indicator of strong panel consensus because: 

• Sixteen of the 17 highest rated items (with median ratings of > 6.00, 
“agree”) displayed IQRs of < 1.61 (indicating whole-group agreement that 
those items were of high importance), and 

• Three of the four lowest rated items (medians < 5) displayed IQRs of < 
1.61 (indicating whole-group agreement that those items were of lower 
importance). 

As suggested by Rayens and Hahn (2000), the IQR may be an insufficient 
criterion for determination of agreement. “Frequency distributions are often used 
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to assess agreement (McKenna, 1994)” (Na, 2006, p. 44), and the criterion of 
some percentage of panelists responding to any given response category is used 
to determine consensus (Loughlin & Moore, 1979, p. 103; Seagle & Iverson, 
2002, p. 1; Putnam, Speigel, & Bruininks, 1995; as cited in von der Gracht, 
2008, p. 53). 

In this study, factors determining consensus included the whole-group IQR and 
frequency of responses at the high end of the scale (respondents choosing scale 
points 6-7) and at the low end of the scale (respondents choosing scale points 1—4). 
These “consensus factors” are shown in Table 2. 

Table 2 

Consensus Factors 

Item importance level Determinants of consensus 


Consensus that an item is of 
higher importance 
Consensus that an item is of 
lower importance 


If IQR < 1.61 and frequency of high scores (6- 
7) > 80% 

If IQR < 1.61 and frequency of low scores (1— 
4) > 25% 


Determining importance (Research Question 2). To determine 
importance, we examined Round 3 panelists’ median ratings for each item. 
Whole-group and subgroup (AEE and CTT) median ratings for each survey item 
were determined using SPSS (Version 22.0) software. The medians were ranked 
using the data ranking function of Microsoft Excel. The ranking indicated which 
of the survey items that the subgroups and the entire panel perceived to be most 
important. Because median ratings for all items were quite high (ranging from 
6.71 to 4.60 on a 7-point scale), the validation panel set the item cutoff point for 
“high importance” at median ratings of < 6.0. No survey items were deemed 
unimportant by the validation panel. 

Determining significant differences (Research Question 3). The Mann- 
Whitney U nonparametric test was used to analyze if intragroup median item 
ratings were significantly different. Nonparametric tests compare medians rather 
than means, and as a result, the influence of outliers is negated (Hayes, 1997). At 
the conclusion of the third survey round, a lack of consensus on any survey item 
reflected sustained differences between the groups in that perceptual differences 
persisted despite the use of the Delphi instrument as a means to develop 
consensus. An alpha level (a) = 0.05 was used for all statistical tests of 
significance. The null hypothesis (H 0 ) was: There is no significant difference 
between AEEs and CTTs in their perception of the importance ofETE concepts 
and skills. A / J -value of < 0.05 on any survey item led to a rejection of the null 
hypothesis for that item. 
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Gap analysis with the Standards for Technological Literacy (Research 
Question 4). in this study, we identified competencies deemed important for all high 
school students to attain as part of their fundamental education. We did a gap 
analysis with the STL to compare survey items rated “important” by the Delphi 
panel to existing benchmarks in the high school level standards. If items were 
similar, rewording of the STL benchmarks based on survey item wording was 
suggested. The validation panel confirmed the gap analysis. 

Findings 

Findings indicated where consensus between the AEEs and CTTs was reached 
about items that were of higher or lower importance. In discussing findings, items 
that were rated highest by the whole group and by each subgroup are identified, 
significant differences between subgroups are illuminated, and potential revisions to 
the STL are suggested. Additionally, findings determined the internal consistency 
(reliability) of the survey instrument and the mean value of the participants’ 
responses with regard to design, modeling, systems, resources, and human 
values. 

Initial survey items were based on the literature review and on recent projects 
probing the importance of ETE concepts. As a result of prelaunch trials, the 
Round 1 survey instrument was revised 11 times prior to first round 
administration as part of a continuous improvement process. 

The response rate to survey Round 1 was 88.6%, and 192 comments were 
received from panelists. Based on panelists’ suggestions, numerous changes 
were made. We attempted to be responsive to all suggestions; however, 
comments were sometimes contradictory, and we chose to accept suggested 
changes in wording that improved the clarity of the item. New items were added 
when two or more experts suggested its inclusion. Sixteen questions were 
reworded, and five new questions were added for the Round 2 survey. 

The number of survey items increased from 32 items in Round 1 to 37 items in 
Round 2. In Round 2, panelists were asked to give high scores sparingly because the 
study was aiming to develop a list of the most essential concepts and skills. The 
response rate was again 88.6%. In Round 2, the IQRs of 13 of 32 items (40%) 
converged, attesting to the efficacy of the Delphi method at driving consensus. 

In the final round, of the 34 panelists who were sent the Round 3 survey, 34 
submitted responses (a 100% response rate). Respondents included 18 AEEs 
(four females and 12 males) and 16 CTTs (three females and 13 males). 
Appendix C presents the median ratings, standard deviations, percentiles, and 
whole-group IQRs by item. Findings are discussed below by research question. 
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Research Question 1 

Where does the strongest consensus exist among the expert panelists 
relative to the importance of specific ETE concepts and skills that all high 
school students in the United States should attain as part of their fundamental 
education? AEE and CTT consensus about high importance was reached on 14 of 
3 8 survey items based on both consensus factors (IQR <1.61 and frequency (6-7) > 
80%) being satisfied. The strongest consensus that items were highly important 
was found on Items R7 and Ml: identify and discuss environmental, health, and 
safety issues involved in implementing an engineering project (Item R7) and use 
representational modeling (e.g., a sketch, drawing, or a simulation) to convey 
the essence of a design (Item Ml). AEE and CTT consensus about lower 
importance was reached on two survey items based on both consensus factors (IQR 
< 1.61 and frequency (1-4) > 25%) being satisfied. The strongest consensus that 
items were of lower importance was found on Items D8 and D12: provide an 
example and an explanation of how design solutions can integrate universal 
design principles to help meet the needs and wants of people of all ages and 
abilities (Item D8) and describe, through an example, how the reliability of a 
system and the risks/consequences associated with its use have or have not been 
adequately considered prior to its implementation (Item D12). A list of items for 
which consensus was reached about higher and lower importance is included in 
Appendix A. 

Research Question 2 

Which ETE concepts and skills does the expert panel perceive to be most 
important for high school students in the United States to attain as part of their 
fundamental education? The ETE concept and skills perceived by the combined 
group to be most important for high school students to attain were: identify and 
discuss environmental, health, and safety issues involved in implementing an 
engineering project (Item R7); use representational modeling (e.g., a sketch, 
drawing, or a simulation) to convey the essence of a design (Item Ml); explain 
why a particular engineering design decision was made, using verbal and/or 
visual means (e.g., writing, drawing, making 3-D models, using computer 
simulations; Item D6); show evidence of considering human factors 
(ergonomics, safety, matching designs to human and environmental needs) when 
proposing design solutions (Item HV6); and safely and correctly use tools and 
machines to produce a desired product or system (Item R4). Panelists’ perceptions 
of the most important ETE items for high school students to leam, by whole-group 
median ratings and rankings, are included in Appendix B. 

Research Question 3 

Where are there significant differences between academic engineering 
educators ’ and classroom technology teachers’ perceptions of the importance of 
ETE concepts and skills? Data analysis using the Mann-Whitney U test indicated 
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that subgroup ratings were significantly different on four survey items at the p < 
0.05 level (see Table 3). All of these items except the third (Item S5) were rated 
higher by AEEs than by CTTs. Not surprisingly, engineers, more than teachers, 
emphasized applying science and mathematics to the solution of design 
problems. 

Table 3 

Significant Differences in Median Item Ratings Between AEEs and CTTs Based 
on the Mann- Whitney U Test 

Median 


AEEs CTTs w , . p-value 

(n = (n= 1 n6 ^ df exact sig. 

Item Survey wording of item 18) 16) U value (2-tailed) 


D2 Solve engineering design 
problems by identifying and 
applying appropriate science 
concepts. 

Dll Provide examples of how 
psychological factors (e.g., 
bias, overconfidence, human 
error) can impact the 
engineering design process. 

55 Explain the difference 
between an open-loop control 
system and a closed-loop 
control system and give an 
example of each. 

56 Develop and conduct 
empirical tests and analyze 
system and analyze test data to 
determine how well actual 
system results compare with 
measurable performance 
criteria. 


6.35 

5.80 

81.00 

33 

.012 

5.27 

4.69 

91.00 

33 

.049 

5.17 

5.85 

88.50 

33 

.040 

6.21 

5.36 

89.00 

33 

.046 
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Research Question 4 

Which concepts and skills that academic engineering educators and 
classroom technology’ teachers agree are highly important and are not presently 
addressed by the STL? The validation panel suggested that survey items with 
median ratings of > 5.70 be considered for inclusion in the next iteration of the 
STL. Recommendations are made that the next iteration of the STL add, 
substitute, or reword standards based on 16 survey items that panelists agreed 
are highly important for high school students to attain as part of their 
fundamental education but are not presently addressed by the STL. Proposed 
changes to the STL are included in Appendix D. 

Most STL benchmarks were written in terms of what students should learn; 
in this study, survey items were written in terms of what students should be able 
to do. Survey items might thus provide additional clarity to teachers and 
curriculum developers relative to measurable performances that would define 
important student capability. As an example, the present STL Standard 2Z 
indicates that students should know that: “Selecting resources involves trade¬ 
offs between competing values, such as availability, cost, desirability, and 
waste” (1TEA, 2007, p. 42). However, in the suggested additions, students 
should be able to: 

• Improve an engineering design by identifying, making, and evaluating 
tradeoffs (D4); 

• Give an example of and investigate the impact of a tradeoff a company 
might make between profitability and environmental, health, or safety 
concerns (HV4); and 

• Engage in a group problem-solving activity to creatively generate several 
alternative design solutions and document the iterative process that 
resulted in the final design (D9). 

Thus, students would be demonstrating their understanding of the above 
benchmark. 

Additional Findings 

Additional findings related to psychometric properties of the survey 
instrument (internal consistency reliability) and to comparing mean scores for all 
items within each of the five domains (subscales) of design, modeling, systems, 
resources, and human values. 

Reliability. Often, Cronbach’s alpha is used when investigating the reliability 
of instruments using continuous or interval data. However, because this study’s 
data results from panelists’ responses to items rated on a Likert scale (scale 
points 1-7), data is ordinal; therefore, an ordinal alpha index of reliability was 
used as an alternative. Reliability coefficients for each subscale were determined 
using statistical methods better suited to ordinal data analysis. 

The SPSS Categories procedure CATPCA (a nonlinear Categorical Principal 
Components Analysis) uses optimal scaling to statistically transform ordinal data 
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into a quantitative numerical variable (Meulman, Van der Kooij, & Heiser, 2004). 
CATPCA provides an ordinal alpha reliability measure, and the reliability 
coefficient calculated is for the transformed variables (IBM Support, 2013). 

To compare and confirm reliability statistics, both Cronbach’s Alpha and 
CATPCA ordinal alpha analyses were conducted (using SPSS, Version 22.0), and 
the results are shown in Table 4. Alpha reliability coefficients normally range 
between 0 and 1. “A reliability coefficient of 0.70 or higher is considered 
‘acceptable’ in most social science research situations” (UCLA, Institute for Digital 
Research and Education, 2017, An Example, para. 2; see also George & Mallery, 
2003; Kline, 1999). It is not surprising that the values for ordinal alpha were higher 
(because ordinal data is being analyzed) than those for Cronbach’s alpha, which 
treats Likert scale data as interval data. 

Mean values of responses by category. Although participants’ answers to 
individual survey items are on an ordinal (Likert) scale, the answers to a group of 
items in a category can be regarded as close to normally distributed interval data. 
Therefore, these data were analyzed using mean values. A comparison of the means 
of each subgroup by category is shown in Table 4 and is also displayed graphically 
in Figure 1. 


Table 4 

Mean Values of AEEs (n = 18) and CTTs (n =16) Final Round Responses Related 
to the Five Categories in the Questionnaire (Scale Points 1-7) 


Category Number AEEs 

of items 


CTTs 


Cronbach’s 

Alpha 


CATPCA 

Ordinal 

Alpha 




Mean 

SD 

Mean 

SD 



Design 

12 

5.8102 

.63517 

5.5885 

.50412 

.783 

.857 

Modeling 

6 

5.5926 

.98389 

5.6458 

.62620 

.773 

.877 

Systems 

6 

5.5926 

.82490 

5.7083 

.40597 

.595 

.728 

Resources 

7 

6.1429 

.64635 

6.2589 

.53253 

.623 

.810 

Human 

7 

5.6825 

.90159 

5.5357 

.53579 

.794 

.917 

values 
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Design Modeling Systems Resources Human 

Values 


■ Mean CTTs 

■ Mean AEEs 


Figure 1. Mean scores for all items on each subscale by subgroup. 


The highest mean scores for both subgroups were obtained in the Resources 
category. The lowest mean score for CTTs was in the Human Values category, 
and the lowest mean scores for AEEs were in the Systems and Modeling 
categories (tied). 

In summary, salient findings included: 

• Descriptive statistics including median ratings, standard deviations, and the 
Interquartile Range (IQR) for each item; 

• A ranked analysis of the engineering and technology concepts and skills 
perceived to be most important for the general education of high school 
students by whole-group median rating; 

• An identification of items for which differences between subgroups were 
statistically significant; 

• A list of concepts and skills that experts agree are highly important for high 
school students to attain as part of their fundamental education that are not 
presently addressed by the STL; and 

• Internal consistency reliability measures of the subscales. 


Limitations 

A limitation of the present research related to the selection of the expert 
panelists: there was a considerable imbalance between more experienced 
(presumably older) and less experience (presumably younger) panelists. Thus, 
perspectives of younger educators who might have reflected more contemporary 
views of the importance of certain ideas and skills may not have been 
adequately considered. Therefore, it is recommended that in selecting panelists 
for future studies, targeted efforts should be made to recruit younger panelists to 
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determine if their perceptions about the importance of knowledge and skills 
related to contemporary technologies differ significantly from their more 
experienced, presumably older, colleagues. 

Conclusions 

Because engineering and technology education are receiving greater 
attention as components of STEM education, support for the establishment of 
PreK-12 ETE programs in the United States has been rapidly growing. Although 
university level academic engineering educators are an ideal professional 
constituency to ally with and support secondary school ETE programs, prior to 
this study, it was uncertain whether they held similar perceptions about the 
fundamental knowledge and skills that high school graduates need for life in a 
technological world to the classroom technology teachers who develop curriculum 
and deliver secondary school ETE instruction. 

We have examined the alignment of the two constituencies’ perceptions 
about the importance of key concepts and skills that all high school students in 
the United States should learn as part of their fundamental education. Our 
findings demonstrate that there is indeed a greater degree of concordance than 
there are perceptual differences between the two constituencies. 

From a theoretical perspective, this research contributes to the body of 
knowledge about the most salient ideas and skills that students need to learn and 
understand in five overarching domains of engineering and technology that are 
repeatedly referenced in the literature: design, modeling, systems, resources, and 
human values. Additionally, this study provides the first research-based 
comparison of perceptions about important ETE ideas and skills between two 
constituencies whose missions focus on preparing students to succeed in our 
technological world through engineering and technology education. 

From the methodological perspective, this study illustrates how the Delphi 
technique can be employed within an education research study in which the 
emphasis is on eliciting and comparing the perceptions of different groups of 
experts. On one hand, the Delphi technique was utilized to identify perceptual 
differences between expert groups with different backgrounds; on the other 
hand, it was used to bridge differences in background in order to forge 
consensus. The Delphi research methodology used in this study was modified 
from the classical Delphi approach in several ways. Modifications that could be 
considered by other researchers include: (1) beginning the Delphi process with a 
set of carefully preselected items that were drawn from the literature review, (2) 
adding validation panel reviews and meetings to help identify panelists and 
initial survey items and to reach post-survey consensus, (3) establishing a set of 
selection criteria for choosing expert panelists, (4) including open-ended text 
boxes to solicit and present arguments for or against items being included in the 
list of “important” survey items, (5) establishing an 1QR range on a Likert scale 
as being indicative of strong consensus, and (6) establishing frequency 
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distribution percentage criteria for responses at both the high end and the low 
end of the scale. 

Within the framework of this research study, a method for examining 
internal consistency reliability suitable to interpreting ordinal data is proposed 
based on Categorical Principal Components Analysis (CATPCA), as a 
replacement for the Cronbach’s alpha coefficient that is typically used to 
interpret interval data. 

From a practical perspective, this research contributes to engineering and 
technology education by: 

• Establishing a basis for educators to develop local, state, and national 
ETE curriculum frameworks, instructional materials for students and 
teachers, and assessments of teaching and learning; 

• Informing a revision of the Standards for Technological Literacy, 

• Elevating the status of school-based engineering and technology 
education by improving the rigor and robustness of curriculum and by 
increasing the advocacy of university faculty members and engineering 
educators; 

• Guiding the design of proposals to foundations and government 
agencies to fund improvement of ETE curriculum and instruction. 
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